Is"Better Data"Better than"Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)

We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-criteria tests while (b) fixing the weaker regions of the training data (using SMOTUNED, which is a self-tuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions. When applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements were independent of the classifier used to predict for quality. We hence conclude that, for software analytics, (1) data pre-processing can be more important than classifier choice, (2) ranking studies are incomplete without pre-processing and (3) SMOTUNED is a promising candidate for pre-processing. Keywords—Search based software engineering, defect prediction, classification, data analytics for software engineering, SMOTE, imbalanced data, preprocessing